Add Scala Spark ETL example to Java-SDK e2e tests #68939
Conversation
## Why Demonstrate and regression-test that the Java SDK can run a real Scala + Apache Spark workload, with task logs routed into Airflow via Log4j 2. ## What - Add `java-sdk/scala_spark_example`: a standalone Scala + Spark 3.5 (local mode) ETL bundle whose three tasks pass scalar results over XCom and log through Log4j 2 (`airflow-sdk-log4j2`). - Run it inside the existing `java_sdk` e2e via a second coordinator and queue (`scala-jdk` / `scala`) with its own `jars_root`, keeping the Java example bundle Spark-free. - Pin the e2e worker JRE to Java 17 and pass Spark's `--add-opens` JVM args. - Add `TestJavaSDKScalaSparkExample` asserting the tasks succeed and the XComs match the fixed dataset (5 rows, total revenue 1000).
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-3-test
jason810496
added
AIP-108: java-sdk
and removed
backport-to-v3-3-test
The e2e test asserted only the extract and load XComs, so the aggregation stage in the middle of the pipeline could regress without the test noticing. Assert its XCom as well, drop the unused dataset constants, and note why the transform reads the upstream count it does not reuse.
The Scala Spark coordinator launched the bundle JVM with a hand-curated subset of Spark's Java 17 module openings. Spark normally injects its full default set through its own launcher, which the raw JavaCoordinator launch bypasses. The subset is enough for the toy aggregation but omits openings that real Spark code paths need (Kryo reflection, off-heap cleaner, charset decoding, Kerberos), so the example would mislead anyone copying it for a non-trivial Spark workload. Mirror Spark 3.5.8's full default module option set instead.
|
We should proably restructure the example project layout to avoid too many free directories lying around. I’ll work on that after this is merged. |
Sure, I explicitly split the scala one out of existing example project as the spark scala one will add unnecessary scala deps to the existing one. |
Review feedback on the Scala Spark e2e example asked to make the bundle
source layout consistent -- it mixed the Maven-standard src/main/resources
with flat src/scala and src/resources -- so the whole tree now follows the
standard src/main/{scala,resources} convention. It also asked that the JDK
module options Spark needs under the Java SDK be documented where users can
find them, not only buried in the e2e test, and that comments restating the
code be trimmed.
phanikumv
left a comment
phanikumv
left a comment
There was a problem hiding this comment.
Non-blocking: the java_sdk e2e job doesn't cache .m2/.gradle in CI, so this second Gradle build pulls Spark's full dependency tree on every canary / java-sdk-touching run.
A separate actions/cache PR is fine if it's worth it.
airflow-e2e-tests/tests/airflow_e2e_tests/conftest.py#L352-L376
Good point, I will raise follow-up PR for this and speed up the build steps as well there. I will merge this one first to ensure the Scala Spark compatibility with Java SDK first to avoid the further regression. |
3 participants
Why
Demonstrate and regression-test that the Java SDK can run a real Scala + Apache Spark workload, with task logs routed into Airflow via Log4j 2.
What
java-sdk/scala_spark_example: a standalone Scala + Spark 3.5 (local mode) ETL bundle whose three tasks pass scalar results over XCom and log through Log4j 2 (airflow-sdk-log4j2).java_sdke2e via a second coordinator and queue (scala-jdk/scala) with its ownjars_root, keeping the Java example bundle Spark-free.--add-opensJVM args.TestJavaSDKScalaSparkExampleasserting the tasks succeed and the XComs match the fixed dataset (5 rows, total revenue 1000).Was generative AI tooling used to co-author this PR?